1. Field of the Invention
This invention relates improvements in software and hardware design verification. More particularly, this invention relates to methods and systems for centrally managing the composition and execution of test suites for verification of different hardware and software.
2. Description of the Related Art
The meanings of acronyms and certain terminology used herein are given in Table 1. The terms Sun, Sun Microsystems, Java, J2EE, J2ME, J2SE, and the Sun logo are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States of America and other countries. All other company and product names may be trademarks of their respective companies.
TABLE 1APIApplication programming interfaceCLDCConnected, limited device configuration. CLDC issuitable for devices with 16/32-bit RISC/CISCmicroprocessors/controllers, having as littleas 160 KB of total memory available.HTTPHyperText Transfer ProtocolIDIdentifierIPInternet ProtocolJ2EEJava 2 Enterprise EditionJ2MEJava 2 Micro EditionJ2SEJava 2 Standard EditionJADJava application descriptorJARJava archiveJDTSJava Device Test Suite execution frameworkMIDletA MIDP applicationMIDPMobile information device profile. A set of JavaAPIs, which, together with the CLDC, provides acomplete J2ME application runtime environmenttargeted at mobile information devices.PlatformAn underlying system on which applicationprograms can run on a computing device. Ittypically includes an operating system, and caninclude supporting hardware, which may be eitherstandardized, or customized for a particularcomputing device.
Tools have been developed in recent years to aid in the design verification of hardware and software systems, for example software suites, hardware circuitry, and programmable logic designs. In order to assure that the design complies with its specifications, it is common to generate a large number of input or instruction sequences to assure that the design operates as intended under a wide variety of circumstances. In general, test systems produce a report indicating whether test programs have been passed or failed, and in some cases may even indicate a module that is estimated to be faulty.
Conventionally, test systems employing complex test suites use a computer-implemented testing framework for computing devices, such as mobile information devices, and for testing software designed to run on such devices. A developer submits a computing product under development, typically a computing device or software that is designed to run on the device to the test system, which runs a selected battery of test programs on the product while monitoring its behavior. Each product under test requires an instance of an execution test harness, or the use of a stand-alone test execution API. The latter is described in commonly assigned application Ser. No. 10/347,748, entitled “Generating Standalone MIDlets from a Testing Harness”, which is herein incorporated by reference.
In environments where testing of a product is ongoing, different aspects may be tested by different teams. As test results are evaluated, it is often necessary to revise the product under test, or to modify the test suites themselves. In such an environment, communicating such revisions to the different testing teams, maintaining version control, synchronization among the teams, and generally coordinating the testing activities is a difficult management problem. Errors could result in inappropriate testing and the wastage of valuable time and testing resources. Indeed, failure of coordination could result in the release of an inadequately tested product into the marketplace. A related problem when many test suites are being concurrently executed is the effort of setting up each test suite with its own test harness and environment. Bundling the test harness with the test suite is not a good solution, as the effort in maintaining up-to-date versions becomes formidable as the number of concurrently operating test suites increases and when the product or the test suites are frequently modified by different test teams.
There is also a need to further customize the composition of established test suites in a particular session, and in all sessions and clients. For example, a bug may have been discovered in a test or group of test programs comprising a test suite. The system administrator may therefore wish to exclude such test programs from performance by the clients of the test execution harness.